Benchmarking is a process for assessing the relative performance of a computer system by running a number of tests against it. Benchmarking is usually associated with assessing performance characteristics of computer hardware, for example, a CPU, and computer software. Software benchmarks are, for example, run against compilers or database management systems. Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures.
Benchmarks are designed to mimic a particular type of workload on a component or system. Current benchmarking techniques do not use real-world data to benchmark systems. Such benchmarking techniques may not provide accurate performance results of the systems, and often, the performance of the system when deployed into a production environment to handle real-world data varies from, e.g., is inferior to, the performance determined by the benchmarking techniques.
Some benchmarking techniques do use real-world data to overcome the above problem of inaccurate performance results. However, such benchmarking techniques typically employ inefficient benchmark generation techniques and are difficult to use. For example, such benchmark generation techniques require physical access to a production server, e.g., a server that handles live traffic, to obtain the real-world data. Obtaining physical access to the production server can sometimes be impossible and/or inconvenient and can result in a significant delay in obtaining the production data. Further, if the benchmarks may have to be generated repeatedly for various production data, e.g., at various times of the day, obtaining the production data is not only challenging but can be an impossible task.
Further, a format in which the performance results are generated by current benchmarking techniques is often not easy to interpret. They also may not provide a comparison between the performances of various systems that are benchmarked. Such representation of the performance results may not facilitate in making informed decisions in selecting one system over the other.